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The conceptual basis for using individual expectancy 
scores as a means for determining the success of instruction is 
presented. An expected gain score is computed for each student, based 
on a pretest score. The assumption is made that without a change in 
the learning environment the student will continue to gain at his 
previous average rate. The data are displayed in an achievement 
idiograph, which shows individual grade placement based on the 
pretest, expectancy score, and posttest placement. Data can De used 
at the individual student level by classroom teachers or aggregated 
by classroom, school, or district to serve the needs of other 
decision-makers. Less than expected gain can serve to quickly draw 
attention to areas of weakness in the curriculum; more than expected 
gain can be considered as "success." Idiographic analysis eliminates 
the necessity for control groups and historical data ds means for 
assessing program success. (Author/KM) 
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IDIOGRAPHIC ANALYSIS OF ACHIEVEMENT MEASURES 



M. L. Rapp*'* 
S. A. Haggart 

The RAND Corporation, Santa Monica, California 

INTRODUCTION 

One of the problems that has plagued the evaluation of compensa- 
tory programs has been establishing the significance of obtained 
achievement gams. Tradit ional design calls for random assignment of 
students to experimental and control groups. But compensatory educa- 
tion is desicned to meet the specific needs of identified groups of 
students. Therefore, it is inappropriate to randomly assign students 
to treatments or non-treatment groups. Even if a control group were 
available outside the program, the assumption of randomization that 
lies behind the tests of statistical significance most commonly used 
is violated. In addition, it is often difficult, if not i mposs i bl e, to 
find a group of comparable students who are receiving a "pure", tra- 
di t ional education. 

A solution often used is to rely on historical data and lo mea- 
sure the effect of some treatment against the results achieved v/ith 
like groups of students in previous years who did not receiv*- special 
programs. Average gains for these groups can be used as an expectancy 
prediction of future gains by like groups if they receive no special 
treatment. Standardized tests are normed to show ten months growth 
for an instructional year. Compensatory programs generally deal with 
a population gaining from four to seven months in a ten-month ins •'•jc- 
tional year. Therefore, if a treatment raises that gain to an average 
of ten months, we can attribute the additional gain to the treatment 
with a reasonable degree of confidence. 

Any views expressed in this paper are those of the authors. They 
should not be interpreted as reflecting the views of The Rand Corpora- 
tion or the official opinion or policy of any of its governmental or 
private research sponsors. Pavers are reproduced by The Rand Corpora- 
tion as a courtesy to members of its staff. 

This concept was developed during the course of our evaluation 
of Project R-3 in San Jose Unified School District as a management tool 
for the project staff. Project R-3 is a demonstration program funded 
by the California State Department of Education under Assembly Bill 938. 
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If the population under consideration is heterogeneous, and there 
is a wide spread of pre-test scores, historical data for that group 
are inappropriate. We have been studying over the last two years a 
total grade population in which a large proportion of students have 
academic deficiencies, but many of the students score at or above 
grade-level on pre-tests. The range of their scores on the pre-test 
at the start of the eighth grade was from the second to the tenth 
grade in both reading and mathematics. 

Our solution is to use individudL expectancy scores, based on 
participants* pre-test scores, and to couple them with individual 
achievement scores. This paper details the conceptual basis for an 
idiographic analysis of these expectancy and achievement scores and 
illustrates its application. 

DERIVING EXPECTANCY SCORES 

We have used a form of expectancy data that is derived from data 
about each of the students participating in the program. We make the 
assumption that a student's pre-test given before the special program 
starts represents the average rate of gain he has made during his pre- 
vious years of schooling. Since we are dealing with an entering eighth- 
grade population (the first year of a two-year longitudinal analysis ^ 
of a continuing program) we divide the entering grade-equivalent score 
for each student by eight to obtain his average rate of gain over the 
previous years (K through 7) he has been in school. We then assume 
that with no special treatment he will continue to make the same average 
gain in future years. 

We would be cautior5 about using this measure in the first two or 
three grades because there would not be sufficient opportunity for 
random variations in learning rate to average out. While we do not 
assume a constant rate of gain over the years for an individual student, 
we consider the average rate of gain over the past years to be a reason- 
able predictor of the average gain during future years and in this 
case for the next two years. We are also assuming that a child had a 
year of kindergarten since the expected score for a large population 
entering first grade is 1.0. 
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If there is a difference between his expected gain and his achieved 
gain at the end of the p rog ram ^rt seems reasonable to attribute that 
difference to the effect of the program in which he was participating. 
A t-test on the difference between the expected and observed means 
can be used to determine whether the difference is significant. 

Since the relationship between entering achievement score and ex- 
pected gain is linear, it is easy to construct a Udo-year expected cjain 
graph using^the formula: 

entering score ^ ^ 
expected gain = g x z 

The entering score is read off the x axis, and the expected score at 
the end of two years off the y axis. An al teniati ve, of course, would 
be to construct a table of expected scores using the same formula. For 
each student we computed an average gain over the past eight years. 
Based on t.his, an expected two-year gain for each student in each sub- 
ject was derived. 

OBSERVED GAINS 

There were 113 students for whom we had eighth-grade (October 1970) 
pre-tests and ninth-grade (June 1972) post-tests in both reading and 
mathematics. Tables 1 and 2 show the number of students in each range 
of months gained in reading and mathematics, respectively. 

Table 3 summarizes the expected and observed gain for the group 
as a whole. 

A t-test was done on each set of scores, reading and math. For 
reading, the difference is significant wel 1 beyond not only the .001 
level, but beyond any tabulated significance level. For math, the 
difference is significant at the .001 level. 

We consider this to be good evidence that both programs made real 
changes in the rate of achievement gain for the program students. 



Table 1 

EXPECTED AND OBSERVED TWO-YEAR GAINS IN READING 



Number of Students 



Months 






Ga i ned 


Expected 


Observed 


<1 


0" 


6 


1-5 


3 


5 


6-10 


i»8 


8 


11-15 


38 


9 


16-20 


15 


15 


21-25 


8 


■ 8 


26-30 


1 


12 


31-35 


0 


18 


36-i»0 


0 


13 




0 


9 




0 


5 


51-55 


0 


i> 


56-60 


0 


1 



It is an artifact of the process for esti- 
mating expected gain that no losses can be pre- 
dicted. 



Table 2 

EXPECTED AND OBSERVED TWO-YEAR GAINS IN MATH 
Number of Students 



Months 



Ga i ned 


Expected 


Observed 


<1 


0" 


i» 


1-5 


1 


12 


6-10 


3^ 


17 


11-15 


'»3 


22 


16-20 


26 


13 


21-25 


8 


18 


26-30 


1 


9 


31-35 


0 


11 


36-i»0 


0 


3 


i»l-i»5 


0 


3 


'(6-50 


0 


0 


51-55 


0 


0 


56-60 


0 


1 



It is an artifact of the process for esti- 
mating expected gain that no losses can be pre- 
dicted. 
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Table 3 



SUMMARY EXPECTED AND OBSERVED DATA 



Reading 
Math 



Expected Gain Observed Gain >• 

1.23 2.60 
1.35 1.7^ 



ACHIEVEMENT IDfOGRAPHS 

In order to visually display these data in a way useful as a manage- 
ment tool, we constructed the achieveiTient idiographs for reading and 
math shown in Figs. 1 and 2. The x axis shows individual students. 
The y axis sh'ows grade levels. The idiograph shows the pre-test grade 
level, the expected gain and the post-test grade level. 

It is possible at a quick glance to ascertain which individual 
students are making better than or less than expected gains. If names 
were attached to each student's record, the measure of success for 
individual students could be seen. This information could be used for 
planning individualized programs for each student. If he is progressing 
well, perhaps no changes are needed. I f he is progressing at the ex- 
pected no-treatment level, or only slightly better than expected, per- 
haps a change in his instructional program is indicated. 

If we look at the achievement idiograph for mathematics we can 
readily see that the program is not reaching students with the higher 
entering scores. This suggests that either the materials are not 
suited to the students or that the style and content of the presentations 
are not suitable. In contrast, reading did very well for the higher 
ability students, but did not achieve uniformly good results with stu- 
dents whose entering scores were at about the trhird grade level. 

The teacher or administrator can pose many questions related to 
the achievement of the group under consideration and quickly have an 
answer by looking at the idiograph. For example, he m^ght want to 
know how many students are at any given grade level, or how many stu- 
dents who entered at some specified grade level left the program at 
the tenth grade level. In addition, by drav/ing vertical lines in the 
appropriate places on the Idiograph, a class may be divided into 
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quartiles, and an assessment of the achievement of different ability 
groups made. These kinds of information are needed as a basis for 
improving the curriculum for individual students as well as for groups 
of students with similar needs. 

USES OF THE ACHIEVEMENT IDiOGRAPH 

Idiographic analysis provides the school staff, administrators and 
teachers, with a useful tool for increasing the effectiveness of educa- 
tion. We have presented some ways in which the teacher might use i ^, to 
modify curriculum and adjust a program to the abilities of individual 
students in his class. 

If teachers are to be Ae\d accountable for the outcomes of their 
students, idiographic anajysis provides a fair basis for making judg- 
ments about the.i r success in relation to the ability of their students. 
A teacher whose students make better than expected gains would objec- 
tively be deemed successful; the teacher whose students made less than 
expected gains would have the information necessary to iiTiprove his 
curriculum. In ord-^r for the data to be of maximum use to the teacher, 
it must be available on a timely basis. This should present no real 
problem because, given pre- and post-test scores and a table of expected 
scores, the idiograph is quickly constructed. 

While the idiograph is a useful tool at the classroom level, to 
be useful at other levels in the school district, or at the state 
level, ic needs to be adapted to higher levels of aggregation. 

In the idiographic analysis of outcome the conceptual basis is 
the same for classrooms, schools and districts as that developed in 
this paper for programs. This is shown in Fig. 3. 

As can be seen from Fig. 3 the individual entities in idio- 
graphs represent higher levels of aggregation. Plotted on the illustra- 
tive idiographs are average scores for the appropriate level of aggre* 
gation. Such idiographs can be constructed for any subject at any 
grade level. It should be borne in mind that as the level of aggrega- 
tion increases from the individual the detail necessary for curriculum 
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improvement is lost. The purpose of using idiographic analysis at the 
higher levels is as an indicator of achievement. Effective or ineffec- 
tive classrooms or schools are readily pinpointed. Where program im- 
provement is indicated, further idiographic analysis can be done at a 
finer level of detai 1 . 
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Fig. 3— Illustrative achievement idio.raphs 
for different levels of aggregation 



If the relationship between socio-economic status and achievement 
is as strong as demonstrated, the expectancy scores of different schools 
will be quite different. Their achievement can then be compared in 
relation to their own population. It removes the necessity for pub- 
lishing data about the socio-economic characteristics of each school; 
that information is inherent in the expectancy score. 
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Finally, idiographic achievement analysis could e compare 

across programs. Title I programs could present their data in this 
fashion and their success be judged in relation to what can reasonably 
be expected in terms of improving student achievement. Demonstration 
programs could also be compared in terms of how wel 1 they do in rela- 
tion to expectancy for their populations. 

Jn this mode of idiographic achievement anal ys is, each program's 
achievement parameters, the pre-test score, the expectancy score and 
the post-test score, would be presented b\ quartile. This would enable 
the State Department of Education to be selective in their recommenda- 
tions about which demonstration programs can be effectively replicated 
for different student populations. 

In short, idiographic achievement analysis provides a basis for 
evaluating the success of any educationj] endea^/or in realistic terms. 
It provides a substitute fur absolute standards of performance and 
measures progress in terms of achievement above expectancy, while at 
the same time it provides a tool lo be used for educational improvement. 



